Marketing Research

MKTG 440

Research Designs

Agenda

This week’s topics

  • Questions from last time?
  • End of today’s class: Pitch day!
  • Research types
    • Qualitative vs. quantitative
  • Research methods for different objectives
    • Exploratory
    • Descriptive
    • Causal
  • Surveys
  • Experiments and validity
  • Lab vs. field experiments

Research Types

Qualitative vs Quantitative Research

First, a high-level distinction between two broad types of research methods:

  • Qualitative research uses data from open-ended, probing questions, usually on small samples.
  • Quantitative research uses formal questions and pre-determined response options, usually collected from a large number respondents.

The data from these research methods typically differ in format:

Qualitative data

What is your evaluation of the front of the vehicle?

“The front of the car looks like an angry face with the headlights as eyes.”

Quantitative data

What is your evaluation of the front of the vehicle? (1: unappealing; 5: appealing)

1      2      ③      4      5

Key Differences

Factor Qualitative Methods Quantitative Methods
Goals Discovery of new ideas and thoughts
Preliminary understanding of relationships
Probe hidden psychological and social processes
Validation of facts, estimates, relationships
Research objectives Exploratory Descriptive and causal
Type of questions Open-ended, unstructured, probing Mostly structured
Time to execute Relatively short time frame Typically significantly longer
Representativeness Small samples Large samples with a focus on sampling
Type of analysis Debriefing, subjective, thematic analysis Statistical, descriptive, causal
Generalizability Often limited Generalizable if done correctly
Necessary skills Interpersonal communications, observation
Interpretation of text or visual data
Statistical analysis, numerical interpretation

Research Methods for Different Objectives

Exploratory Research

Exploratory research

Exploratory research: Research used to generate ideas, clarify concepts and the nature of the problems, and to establish research priorities.

  • Get ideas!
  • Define terms and concepts
    • Ex: What is Meta’s brand image?
  • Uncover future research questions
    • Ex: How do people feel about spending a non-trivial amount of time in a Metaverse – a virtual world where people socialize, play, and work?
  • Ex: Establish research priorities
    • Customers are complaining about changes to the feed? What, if any, immediate actions should be taken?
  • Typically takes place first (before descriptive or causal research)

Direct (nondisguised) methods

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '24px'}, 'flowchart': {'nodeSpacing': 80, 'rankSpacing': 80, 'htmlLabels': true, 'useMaxWidth': false, 'padding': 8}}}%%
flowchart TB
    A["Exploratory Research<br>Methods"] --> B["Direct<br>(Nondisguised)"]
    A --> C["Indirect<br>(Disguised)"]
    B --> D["In-Depth<br>Interviews"]
    B --> E["Focus<br>Groups"]
    C --> F["Projective<br>Techniques"]

    style B fill:#1a4480,stroke:#1a4480,color:#fff
    style C fill:#fff,stroke:#000
    style A fill:#fff,stroke:#000
    style D fill:#fff,stroke:#000
    style E fill:#fff,stroke:#000
    style F fill:#fff,stroke:#000

In-Depth Interviews

In-depth interview: an interview where a trained interviewer asks a respondent a set of structured probing questions.

  • Typically one-on-one
  • 30-60 minutes
  • Focused
    • No other participants to impress
    • Few distractions
    • Highly structured - interviewer controls flow
  • Opportunities for more detailed discussion (follow-ups)

Focus Groups

Focus group: A moderated group discussion designed to surface language, motivations, beliefs, and reactions to ideas (e.g., ads, concepts, packaging).

  • Typically 5-10 participants
  • 60-90 minutes
  • Participants are not randomly selected (purposive sampling)
  • Moderator asks questions, stimulates, and controls the direction of the conversation

Links:

Interviews vs. Focus Groups Discussion

What do you think are some of the advantages and disadvantages of focus groups compared to in-depth interviews?

  • Group dynamics (organic discussion vs. groupthink)
  • Sensitive topics
  • Efficiency
  • Representativeness
  • Interviewer presence (observer interference)

Indirect (disguised) methods

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '24px'}, 'flowchart': {'nodeSpacing': 80, 'rankSpacing': 80, 'htmlLabels': true, 'useMaxWidth': false, 'padding': 8}}}%%
flowchart TB
    A["Exploratory Research<br>Methods"] --> B["Direct<br>(Nondisguised)"]
    A --> C["Indirect<br>(Disguised)"]
    B --> D["In-Depth<br>Interviews"]
    B --> E["Focus<br>Groups"]
    C --> F["Projective<br>Techniques"]

    style B fill:#fff,stroke:#000
    style C fill:#1a4480,stroke:#1a4480,color:#fff
    style A fill:#fff,stroke:#000
    style D fill:#fff,stroke:#000
    style E fill:#fff,stroke:#000
    style F fill:#fff,stroke:#000

The power of the mask

“Man is least himself when he talks in his own person. Give him a mask, and he will tell you the truth.”

– Oscar Wilde

Projective Techniques

Projective technique: A form of questioning that encourages respondents to project their underlying motivations, beliefs, attitudes, or feelings regarding the issues of concern…

  • Onto a third party
  • Into a task situation
  • Onto an inanimate object or other vague stimulus

Can be used within focus groups or in-depth interviews:

  • Role playing
  • Word association
  • Sentence completion
  • Cartoon (balloon)

Why Projective Techniques?

What do you think?

  • Direct questioning is sometimes of limited value because people may not be aware of the real reasons for their feelings or choices, and resort to obvious or conventional explanations.

  • Indirect approaches, such as projective techniques, are aimed at uncovering “subconscious” motives or helping people voice hard-to-articulate (or sensitive) ideas

Role playing

Role playing: Respondents are asked to play the role or assume the behavior of someone else.

Classic Example: Nescafe (1950)

  • Context: unexpected customer resistance to Nescafe instant coffee, marketed as a new easy way to make coffee at home
  • When women were questioned directly about why they did not like instant coffee, the typical answer was they did not like its flavor
  • Researchers suspected flavor was a cover story
  • Experiment by Mason Haire [Journal of Marketing, Vol. 14, No. 5 (Apr. 1950), pp. 649-656]

The Nescafe Shopping List Study

100 housewives (it was the 1950s…) were asked to review one of two shopping lists and to role-play as a shopper going to the grocery store to buy the items on the list.

Shopping List 1

  • 2 pounds of hamburger
  • Bunch of carrots
  • 1 can, Rumford’s Baking Powder
  • Nescafe Instant Coffee
  • 2 cans, Del Monte Peaches
  • 5 pounds of potatoes

Shopping List 2

  • 2 pounds of hamburger
  • Bunch of carrots
  • 1 can, Rumford’s Baking Powder
  • Maxwell House Coffee
  • 2 cans, Del Monte Peaches
  • 5 pounds of potatoes

This shopper is:

  • Lazy (48%)
  • Did not plan purchases (48%)
  • Thrifty (4%)
  • Good wife (4%)

This shopper is:

  • Lazy (4%)
  • Did not plan purchases (12%)
  • Thrifty (16%)
  • Good wife (16%)

Nescafe’s Response

Later ads focused less on “quick, efficient, economical”, and instead shifted towards emphasizing social acceptability and quality (“satisfy your coffee hunger”), and portraying instant coffee as something you could “serve to guests with pride”.

Sentence Completion

Sentence completion: A projective technique involving the presentation of incomplete sentences to respondents who are asked to complete them in their own words. The goal is to derive “automatic” connections to stimuli.

Do you think it’s important to give blood?

  • Most people say yes in direct elicitation (i.e., given a numeric rating scale)

Sentence completion task

Respondent 1:

  • I always give blood during blood drives at work, unless I’m sick.
  • People who don’t give blood are pretty selfish in my opinion.

Respondent 2:

  • I always give blood during blood drives at work, unless I’m in a hurry.
  • People who don’t give blood just don’t like needles.

Word Association Test (WAT)

A word-association test is a projective technique where participants are presented with a word or phrase and asked to quickly respond with the first word or idea that comes to mind.

The goal is to uncover subconscious or less-filtered attitudes, feelings, and associations that participants hold toward the stimulus, which might not surface through direct questioning.

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '18px', 'background': '#ffffff'}, 'flowchart': {'nodeSpacing': 20, 'rankSpacing': 30}}}%%
flowchart TB
    N["NIKE"] --> A1["Sports"]
    N --> A2["Shoes"]
    N --> A3["Michael<br>Jordan"]
    N --> A4["Cool"]
    N --> A5["Just<br>Do It"]

    style N fill:#fff,stroke:#000,stroke-width:2px
    style A1 fill:#fff,stroke:#000
    style A2 fill:#fff,stroke:#000
    style A3 fill:#fff,stroke:#000
    style A4 fill:#fff,stroke:#000
    style A5 fill:#fff,stroke:#000

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '18px', 'background': '#ffffff'}, 'flowchart': {'nodeSpacing': 20, 'rankSpacing': 30}}}%%
flowchart TB
    O["ORGANIC FOOD"] --> B1["Healthy"]
    O --> B2["Sustainable"]
    O --> B3["Environmentally<br>friendly"]
    O --> B4["Expensive"]
    O --> B5["Vegetables"]

    style O fill:#fff,stroke:#000,stroke-width:2px
    style B1 fill:#fff,stroke:#000
    style B2 fill:#fff,stroke:#000
    style B3 fill:#fff,stroke:#000
    style B4 fill:#fff,stroke:#000
    style B5 fill:#fff,stroke:#000

Cartoon (Balloon) Test

A cartoon test is very similar to a sentence completion task and serves much the same purpose. Cartoon tests show a cartoon illustration in which one or more characters are present. At least one of the characters has an empty dialogue bubble.

Third-Person Technique

Third-person technique: The respondent is asked to talk about someone else, such as a neighbor or a friend. In this case, the respondent is asked to relate the beliefs and attitudes of a third person rather than directly expressing personal beliefs and attitudes.

Goal: Minimize the social pressure to give a politically correct response

Examples:

  • Which candidate do you think your neighbor is voting for? (Trump or Harris)
    • Compare to “Which candidate are you voting for? (Trump or Harris)”
  • How much do you think other people your age drink on a Saturday night?

Pros and cons of projective techniques

Pros

  • Disguising the purpose of the study allows us to elicit responses that subjects may be unwilling or unable to give otherwise

  • Useful when the issues are personal, sensitive, or subject to strong social norms

  • Useful when underlying motivations, beliefs, and attitudes are operating at a subconscious level

Cons

  • Require highly trained interviewers

  • Risk of interpretation bias

    • When the researcher unintentionally reads their own assumptions, expectations, or preferred story into participants’ ambiguous responses
    • Skilled interpreters are also required to analyze the responses
  • Engage people in unusual behaviors

Descriptive Research

What is descriptive research?

Descriptive research: research targeted at describing the characteristics of an existing marketing situation

  • Questions related to who, what, where, when, how
    • Who buys iPhone 17? When do riders use Uber? Where do consumers buy coffee on campus? How optimistic are people about the economy?
  • Mostly Quantitative

Common data collection approaches:

  • Observational data (secondary)
    • Store audits, consumer panels, TV panels, online panels
  • Surveys and questionnaires (primary)

Observational Data

What is observational data?

Observational data: Observe and record patterns of people, events, or other phenomena

  • Expedia: Rick spent 20 minutes on Expedia searching for vacation packages. After clicking on the pages of five different hotels, he booked a 3-night stay at Hyatt Zilara in Cancun.

  • Apple: Zach watched the entirety of the newest season of Slow Horses in one sitting.

Data can be collected by human observers, cameras, RFID, social media platforms, etc.

Common Industry Data

  • Store audits (retail scanner data)
  • Consumer panels
  • Media panels
    • TV
    • Internet
    • Social media
  • Firm-specific data
    • Social media profiles
    • Website clickstream
    • App behaviors
    • Internet usage from an ISP

Store audit data

Store audits (retailer scanner data) consist of weekly pricing, volume, and store environment information generated by point-of-sale systems from participating retail chains across markets.

  • Tens of thousands of participating grocery, drug, mass merchandisers in major US markets

  • Weekly product data for >2.6 million UPCs (food, drug, liquor, convenience)

    • units (sales), price, feature indicator, and display indicator
  • E.g., Supermarket X sold 200 units of Tropicana (59oz) in week 2 at $3.99

  • Sources: Nielsen, IRI

  • Example questions:

  • How does assortment of category Z vary across different store types (grocery vs. drug vs. mass merchandisers)?
  • What is the effect of in-store displays on sales of product Y?
  • Does our product get better placement (eye-level shelf position, end cap, etc.) than competitors’ products?
  • Demand analysis
  • Competitor intelligence

Consumer panels

Consumer panel: Households provide information about purchases and/or media consumption. The same households are tracked over time.

National Consumer Panel:

  • Nielsen/IRI Panelists (40,000-60,000)
  • Demographic info (e.g., income, size, education, age)
  • Geographic info (county, zip code)
  • Product ownership (e.g., TV, car)
  • Weekly purchase and/or viewing info

Example: Household Z purchased Yoplait from Safeway in Broadway Kino plaza in Week 1, Dannon from a CVS in Week 2, Chobani in Week 3 and Chobani in Week 5.

Example questions:

  • Who buys what, how often, how much, and where?
  • How do purchase patterns vary by demographics (age, income, family size)?
  • Do promotions increase purchase frequency, lead to stockpiling behavior, or induce brand switching?

Other media panels

Media panels (Nielsen, iSpot, TVision):

  • Measure the audience from TVs, computers, and other devices.
    • Average number of viewers for Super Bowl = 100 M

Clickstream data (ComScore):

  • Pages a user visits and the sequential stream of clicks as they move across the web
    • Landing page → product page 1 → product page 2 → review page → shopping cart → transaction

Example questions:

  • What is the reach and frequency of my ad campaign across demographics?
  • How much overlap is there between TV and digital ad reach?
  • Where should we place ads to effectively reach our target audience?
  • Shopping funnel analysis: Does position of the product on landing page affect the likelihood of purchase?

Social media monitoring

Social media monitoring is a way to observe people and how they interact with each other online and within social media

  • Can be perceived as “a focus group of millions of people”

Applications:

  • Millions of “social mentions” of the Fortune 100 companies each month

  • Access all the mentions (of your brand/product) and aggregate the user-generated content across different types of platforms into a single stream of information

Example questions:

  • What are people saying about our new product launch?
  • How do current events beyond our control (e.g., natural disasters, political events) impact brand sentiment?

Cautions: Self-selected sample (are the people posting on platform X representative of the target market?)

Surveys

What is a Survey?

  • Ask respondents for information using verbal or written questions
  • Questions are fixed and structured
  • Respondents are a sample of the researcher’s “target population”
  • Data are mostly numeric (e.g., multiple choice)

Useful when…

  1. Respondents know the answer
  2. We care what they have to say
  3. They are willing to tell us the truth

Types of Survey Methods

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '16px'}, 'flowchart': {'nodeSpacing': 20, 'rankSpacing': 40}}}%%
flowchart TB
    A["Survey Methods"] --> B["Telephone-<br>Administered"]
    A --> C["Personal-<br>Administered"]
    A --> D["Self-<br>Administered"]

    B --> B1["Traditional<br>Telephone"]
    B --> B2["Computer-Assisted<br>Telephone Interviewing<br>(CATI)"]

    C --> C1["In-Home"]
    C --> C2["Mall Intercept"]
    C --> C3["Computer-Assisted<br>Personal Interviewing<br>(CAPI)"]

    D --> D1["Mail"]
    D --> D2["Mail Panel"]
    D --> D3["Online"]
    D --> D4["Drop-off"]

    style A fill:#fff,stroke:#000
    style B fill:#fff,stroke:#000
    style C fill:#fff,stroke:#000
    style D fill:#cce5ff,stroke:#000
    style B1 fill:#fff,stroke:#000
    style B2 fill:#fff,stroke:#000
    style C1 fill:#fff,stroke:#000
    style C2 fill:#fff,stroke:#000
    style C3 fill:#fff,stroke:#000
    style D1 fill:#fff,stroke:#000
    style D2 fill:#fff,stroke:#000
    style D3 fill:#cce5ff,stroke:#000
    style D4 fill:#fff,stroke:#000

In recent years, the vast majority of survey-based marketing research is conducted via online surveys

  • Low cost, speed, ease of targeting, scalable
  • But, some populations may be more difficult to reach

Survey Example: U.S. Streaming Landscape

Whip Media surveyed four thousand SVOD users.

Source: The Streaming Landscape Report

Survey Results: Why Too Many?

Survey Results: Preference for Paying vs. Having Ads

When to use a survey?

Pros

  • Easy - easy to administer to large number of respondents

  • Reliable - standardization reduces variability

  • Efficient - easy coding, analysis, and interpretation of data

Cons

  • Non-response - people don’t always respond

  • Rigid - difficult to capture beliefs or feelings

  • Sensitive - properly wording questions is not easy

  • Sampling - tough to get the right sample

Causal Research Designs

Data-driven decision-making

  • We want know whether we should increase our advertising budget.
  • A junior analyst provides us with the following data.

Monthly Sales (in ’000s) = 1.2 + 0.35 × AdSpend | R² = 0.51

A Causal Relationship

Causal relationship: the relationship between cause and effect.

  • “When you do X, Y will happen”

Correlation is not sufficient

Alternate explanation: Reverse causality

Organic growth in sales leads to larger advertising budget

Alternate explanation: Third factor

Predictable cyclicality in demand + advertising

  • What product might exhibit this pattern?

Correlation vs. causation

Examples in Marketing:

  • Starbucks release the pumpkin spice latte in October. Sales of hot drinks in Starbucks start increasing in October.
    • Does the release of PSLs cause more sales of hot drinks?
  • Product rankings on Amazon, WalMart, BestBuy
    • Do higher ranked products sell better, or do products that sell better get higher ranks? Both?

Three Factors Necessary for Causation

1. Correlation (Concomitant Variation)

  • Evidence of association between X and Y
  • Can be shown with ρ, the correlation coefficient

2. Temporal Antecedence

  • X must occur before Y

3. No third factor driving both X and Y

  • Control of other causal factors

Causation Diagram

A change in X causes a change in Y.

Three Factors Visualized

Correlation

When X changes, Y changes
When Y changes, X changes
(association)

Antecedence

X comes before Y
Y doesn’t come before X

No Third Factor

No third factor, Z, causes the change to both X and Y.

Causation: Combining all 3

Causal Research Designs

Causal Research designs: study the causal effects of changing something in the marketing environment (X) on an outcome of interest (Y).

  • Independent variable (IV): measures what the researcher changed

  • Dependent variable (DV): measures of effects or outcomes that occur as a result of changes in levels of independent variable(s)

Experiments, Randomized Control Trials, Test Markets, A/B Testing

  • The ONLY research designs that rigorously establish causality
  • Difficult part: making sure that change in DV was not caused by something other than IVs (confounding variables, confounders). I.e., no third factor that affects IVs and DV simultaneously

Experimental Design: Treatment and Control Groups

Treatment group

  • Group exposed to the manipulation/treatment
  • X = ORANGE

Control group

  • Group not exposed to the manipulation/treatment
  • X = BLUE

Random Assignment

  • Participants randomly assigned to condition
  • This avoids selection effects
  • Balances individual differences among participants across groups

Netflix Design Example

Netflix wants to understand the causal effects of box art (A, B, C, or D) on the probability of clicking on the show.

How would they set up and run the experiment?

Read more about Netflix A/B testing here.

Experiments and Validity

Experiments differ in their internal and external validity

Internal validity: ability to confidently draw causal conclusions

  • Basically, how clean is the experiment? Is it free of confounds?

External validity: ability to generalize from research setting to other contexts (i.e., the real world)

  • Is the research environment so context-free that results are useless?

Difficult to achieve high internal AND external validity

Example: Internal vs. External Validity

Suppose you want to know what tastes better: Coke or Pepsi

Design A: Unlabeled cups

Design B: Labeled cans

Which design has higher internal validity? Which design has higher external validity?

Experiments and Validity: Confounds

A confound (third factor) is a factor that influences both the dependent variable and independent variable.

Example of a confounded experiment: Suppose you want to learn the effect of container shape on amount of soda consumed. You run the following experiment. You take an unlabeled can and fill it with Pepsi. You take an unlabeled bottle and fill it with Coke. You randomly assign participants to drink from either the can or the bottle, and measure how much soda they consume.

  • What is the DV?
  • What is the IV?
  • What is the confound?

Lab vs. Field Experiments

Lab Experiments

  • Artificial (controlled) setting
  • Lab’s subject pool

Pro:

  • Limited effects from extraneous variables
  • Quick and inexpensive

Con:

  • Not a natural setting (generalization might be a problem)

→ Higher internal validity

Field Experiments

  • Natural setting
  • Supermarkets, online

Pro:

  • Generalizable

Con:

  • Expensive and time-consuming
  • Difficult to control extraneous variables
  • No secrecy

→ Higher external validity

For Next Class

To Do

  • Meet with your team
  • Complete GA1
    • Choose/finalize a research topic
    • Agree on norms for group work
  • Preview GA2
    • Conduct a focus group

Appendix

How to Be a Better Interviewer: RASA

RASA is a simple framework popularized by sound and communication expert Julian Treasure for cultivating better listening and more effective communication:

  • Receive - Be fully present. Actively pay attention to what the speaker is saying. Avoid distractions and give them your undivided focus.

  • Appreciate - Use small verbal (e.g., “I see,” “Mm-hmm”) or nonverbal cues (nodding, eye contact) to let the speaker know you are attentive and value what they’re sharing.

  • Summarize - Recap or paraphrase what you’ve heard to ensure mutual understanding. Phrases like “So, what I’m hearing is…” help confirm you’ve caught the essence.

  • Ask - Invite clarification or deeper insight by asking open-ended questions. This both demonstrates engagement and encourages the speaker to elaborate.

“Unused Coupons Still Pay Off” - HBR

https://hbr.org/2012/05/unused-coupons-still-pay-off

  1. What is the research question?

  2. What did they find?

  3. How does the article intend to affect decisions?

  4. Any potential problems with this study?

Example: Coupon Study

Finding:

Customers who (1) Received coupons and (2) Did not redeem > Customers who did not receive coupons

Is this a good comparison?

Could there be other differences between these groups?

  • Coupon recipients were targeted
  • Composition of the sample on the left can be very different from the right

A potential third factor: targeting variables

“In a experiment with eight national retailers, we analyzed campaigns involving more than 500,000 targeted coupons, for items representing more than 300 brands, mailed out over 16 months.”

What is a Good Controlled Experiment?

Create groups where the only difference between them is the experimental “treatment”

  • e.g., testing the effect of Coupons

Randomized Controlled Trials (RCT)

Treatment Group

Random set of customers are sent coupons

Control Group

Rest of the customers are not sent coupons

Only difference between individuals in (A) and (B) is the coupons!

So…What is the “Real” Effect of Targeted Coupons?

Context: ticket selling website (e.g., Ticketmaster, StubHub)

  • 70 offers, 52k individuals
  • Every offer is run as an (A/B) experiment:
    • The target group is decided for every offer (business as usual)
    • Within the target group
      • A random (small) subset is not sent the offer email
      • The rest get the offer email

Findings:

  • The average expenditure up by 37.2% → Treatment effect
  • 90% of these gains are not through redemption of the offers (long-term effect)
  • Emailed offers → “advertising” for the firm’s products

Sahni, Zou and Chintagunta (2016), “Do Targeted Discount Offers Serve as Advertising? Evidence from 70 Field Experiments”